Methods and apparatus for encrypted indexing and searching encrypted data

ABSTRACT

In some embodiments, an apparatus includes a memory and a processor. The processor is configured to receive an index file that associates a characteristic in a set of documents with a set of information associated with the characteristic in the set of documents. The processor is further configured to generate an index identifier associated with the index file and calculate a set of pseudorandom logical block identifiers associated with a set of storage locations of a database based on the index identifier. The processor is then configured to parse the index file into a set of index data portions and send a signal to the database to write each index data portion from the set of index data portions at a different storage location within the database as indicated by a different identifier from the set of pseudorandom logical block identifiers.

RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.15/259,863, filed on Sep. 8, 2016 and entitled “SYSTEMS AND METHODS FORSTORING DATA BLOCKS USING A SET OF GENERATED LOGICAL MEMORYIDENTIFIERS,” which is incorporated herein by reference in its entity.

BACKGROUND

Some embodiments described herein relate generally to methods andapparatus for indexing and searching encrypted data and/or documents. Inparticular, but not by way of limitation, some embodiments describedherein relate to methods and apparatus for storing encrypted index filesusing a set of generated logical memory identifiers and searching in theencrypted index files.

Enterprises often encrypt their electronic files containing sensitivedata for security and privacy concerns. Searching among a number ofencrypted documents based on content within the encrypted documents canbe challenging. Known solutions include defining an association betweena search string (e.g., a keyword) and a set of encrypted fileidentifiers that identifies a set of encrypted files. Instead ofsearching based on the search string, these solutions involve searchingbased on the set of encrypted file identifiers. These solutions,however, still present a security problem because the associationbetween the search string and the set of encrypted file identifiers isnot encrypted and is open to security attacks.

Accordingly, a need exists for improved and secure methods and apparatusfor indexing and searching encrypted documents.

SUMMARY

In some embodiments, an apparatus includes a memory and a processoroperatively coupled to the memory. The processor is configured toreceive an index file that associates a characteristic in a set ofdocuments with a set of information associated with the characteristicin the set of documents. The set of information includes a documentidentifier (1) from a set of document identifiers and (2) for eachdocument from the set of documents. The processor is further configuredto generate an index identifier associated with the index file andcalculate a set of pseudorandom logical block identifiers associatedwith a set of storage locations of a database based on the indexidentifier. The processor is configured to parse the index file into aset of index data portions and send a signal to the database to writeeach index data portion from the set of index data portions at adifferent storage location within the database as indicated by adifferent identifier from the set of pseudorandom logical blockidentifiers.

In some embodiments, a method includes receiving, from a compute device,a search request including a search string associated with acharacteristic in a set of documents within a database. The method alsoincludes identifying, based on the search string, a set of logical blockidentifiers associated with a set of storage locations of the databaseand retrieving index data stored at each storage location from the setof storage locations using the set of logical block identifiers. Themethod includes compiling a search result file based on the index datastored at each storage location from the set of storage locations andidentifying, based on the search result file, a set of informationassociated with the characteristic in the set of documents. The methodincludes sending a first signal that includes the set of information tothe compute device. The method also includes modifying an initializationvector for the index data stored at each storage location from the setof storage locations to define modified index data for each storagelocation from the set of storage locations. Moreover, the methodincludes sending a second signal to the database to write the modifiedindex data for each storage location from the set of storage locationsto that storage location from the set of storage locations within thedatabase.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of an index searching system,according to an embodiment.

FIG. 2 is a schematic block diagram of a security device within azero-trust storage, according to an embodiment.

FIG. 3 is a flowchart illustrating an initialization process, accordingto an embodiment.

FIG. 4 is a flowchart illustrating an index searching process, accordingto an embodiment.

FIG. 5 is a flowchart illustrating an index file upload process in thecase of a read operation, according to an embodiment.

DETAILED DESCRIPTION

In some embodiments, an apparatus includes a memory and a processoroperatively coupled to the memory. The processor is configured toreceive an index file that associates a characteristic in a set ofdocuments with a set of information associated with the characteristicin the set of documents. The set of information includes a documentidentifier (1) from a set of document identifiers and (2) for eachdocument from the set of documents. The processor is further configuredto generate an index identifier associated with the index file andcalculate a set of pseudorandom logical block identifiers associatedwith a set of storage locations of a database based on the indexidentifier. The processor is then configured to parse the index fileinto a set of index data portions and send a signal to the database towrite each index data portion from the set of index data portions at adifferent storage location within the database as indicated by adifferent identifier from the set of pseudorandom logical blockidentifiers.

In some embodiments, a method includes receiving, from a compute device,a search request including a search string associated with acharacteristic in a set of documents within a database. The method alsoincludes identifying, based on the search string, a set of logical blockidentifiers associated with a set of storage locations of the databaseand retrieving index data stored at each storage location from the setof storage locations using the set of logical block identifiers. Themethod includes compiling a search result file based on the index datastored at each storage location from the set of storage locations andidentifying, based on the search result file, a set of informationassociated with the characteristic in the set of documents. The methodincludes sending a first signal that includes the set of information tothe compute device. The method also includes modifying an initializationvector for the index data stored at each storage location from the setof storage locations to define modified index data for each storagelocation from the set of storage locations. Moreover, the methodincludes sending a second signal to the database to write the modifiedindex data for each storage location from the set of storage locationsto that storage location from the set of storage locations within thedatabase.

In some embodiments, an index searching system, for example, can beconfigured to store data (e.g., index files) and hide the operations ofthe storing and the target of the memory location where the data isstored. In some implementations, for example, an index searching systemis a storage system configured to store data without storing and/ormaintaining, for example, the data's meta-data, content of the data, akey used to encrypt the data, a storage location of the data within thestorage system, a storage operation associated with storing the data,and/or the like.

In some embodiments, an index searching system, for example, can beconfigured to retrieve, based on a search string, a set of locations ina memory where index data are stored and compile a search result filefrom the index data. The index searching system can then be configuredto retrieve document information (e.g., document identifiers) associatedwith the search string from the search result file and send suchdocument information to a client device.

In some embodiments, an “index file” can be, for example, a computerfile or collection of data with an index that allows retrieval ofrelevant information or documents based on a search query. The indexfile can be generated (e.g., by an index server or other compute device)by parsing components (e.g., words, letters, or meta-data) of a set ofdocuments or other form of media and inserting the parsed componentsinto forward or inverted indices. Such document parsing (similar tosearch engine indexing) can involve natural language processing. Theterms “indexing”, “parsing”, and “tokenization” are used interchangeablyherein.

For example, an index file can include a set of index entries. Eachindex entry from the set of index entries defines an association of acharacteristic (e.g., a word, a letter, a phrase, or meta-data) in a setof documents with a set of information associated with thecharacteristic in the set of documents. The set of information caninclude, for example, a document identifier from a set of documentidentifiers for each document from the set of documents, a set oflocations of the characteristic in the set of documents, a set ofstatistical information associated with the characteristic in the set ofdocuments (e.g., the number of times the characteristic appears in theset of documents, the portions of the documents typically including thecharacteristic, etc.), a relationship information associated with arelationship between a first characteristic in the set of documents anda second characteristic in the set of documents (e.g., the proximity ofand/or distance between two words in the documents from the set ofdocuments, the portions of the document including the firstcharacteristic and/or the second characteristic, the type of documentscontaining the first characteristic and/or the second characteristic,etc.), and/or the like.

For example, the index file can include an index entry defining anassociation of phrase “San Francisco” with (1) the document identifierof each document that has the phrase “San Francisco”, (2) the locationin the storage (e.g., backend storage 107 of FIG. 1) of each documentthat has the phrase, (3) the location of the phrase in each document(e.g., page 2, paragraph 3, line 4), (4) how many times the phraseappears in each document, (5) the other words that are in the samesentence of the phrase in each document, (6) the document identifier ofeach document that has the phrase “San Francisco” and the phrase“software engineer,” and/or the like.

The index file can be updated and/or redefined. Additionalcharacteristics and the associations of such characteristics with a setof information can be added to the index file. For example, an indexfile can include two separate index entries. The first index entry inthis example describes an association of phrase “San Francisco” with afirst set of information in the set of documents (e.g., the documentidentifier of each document that has the phrase “San Francisco”). Thesecond index entry describes an association of phrase “softwareengineer” with a second set of information in the set of documents(e.g., the document identifier of each document that has the phrase“software engineer”). A new index entry can be added to the index filethat describes an association of phrase “San Francisco” and phrase“software engineer” with a third set of information in the set ofdocuments (e.g., the document identifier of each document that has boththe phrase “San Francisco” and the phrase “software engineer”). In someinstances, these new index entries can be added to the index file basedon how often these phrases are queried. For example, when the phrase“San Francisco” and the phrase “software engineer” are frequentlyqueried together, the new index entry can be added to the index filesuch that when a client device provides the combined query, a securitydevice can determine storage locations for the combined index entry(rather than for the two separate index entries). This can reduce thecomplexity of frequent searches.

The term “meta-data” refers to information of the data that is notcontained in the content of the data but nevertheless can be kept and/oridentified by a storage system. For example, for a file document, themeta-data can include the file name, file size, file type, filelocation, file author, and/or the like.

The term “document” can refer to, for example, a document, a datastructure and/or a file stored in a storage system. A document can be,for example, an index file, a data file, a video, an image, a webpage, amessage such as email or a chat message, a record in a database system,and/or the like.

The term “storage operations” can refer to, for example, operations oractions supported by a storage system. Typical operations can include,for example, read, write, update, append, delete, search, query, index,and/or the like.

The term “storage” refers to any suitable storage system such as, forexample, a physical storage system or a cloud-based storage system. Astorage includes a memory that allows storage of document(s) that can besubsequently retrieved.

The term “storage unit” refers to a unit of operation used by a storagesystem. For instance, for a file system (e.g., Microsoft® Windows NewTechnology File System (NTFS)), the storage unit can be a file; for adatabase system (e.g., MySQL® database), the storage unit can be arecord or a row; for an object store (e.g., Amazon™ S3), the storageunit can be an object; and for a block-based storage array (e.g.,DataDirect™ Networks), the storage unit can be a block. For a zero-truststorage, internally the storage unit can be a logical block.

The term “pseudorandom number generator” or the term “randomizer module”refers to a module configured to generate a sequence of numbers whoseproperties approximate the properties of a sequence of random numbers.The generated sequence can be pseudorandom because given the same seedthe same sequence can be reproduced. In some instances, the pseudorandomnumber generator can take as inputs, for example, a seed, a range,and/or a cryptographic key. The range can determine (or define) a rangeof output numbers generated by the pseudorandom number generator. Theseed is described in further detail below. In some implementations, thecryptographic key can be predetermined and hidden from any device and/ormodule accessing and/or interacting with the pseudorandom numbergenerator. In other instances, the device and/or module accessing thepseudorandom number generator can supply a cryptographic key to thepseudorandom number generator to customize the process used to generatethe sequence of numbers so that given the same seed and same range, thesequence of numbers can be different if different cryptographic keys areprovided. Similarly, for the same pseudorandom number generator and thusthe same process, if the same cryptographic key, seed, and range areprovided, the same sequence can be generated.

The term “seed” can be a set of initial values (or a single value) thatis provided to a pseudorandom number generator to generate a sequence ofpseudorandom numbers. In some implementations, given the same seed tothe same pseudorandom number generator, the sequence of numbersgenerated can be the same.

The term “encryption” refers to any cryptographic method to encode adocument so that only authorized parties can read/access the encodeddocument. The term “decryption” refers to any cryptographic method todecode a document so that the encoded document can be decoded and read(or accessed) by authorized parties, devices, modules, and/orcomponents.

The term “initialization vector” refers to any numerical ornon-numerical value. For example, in some instances, an initializationvector can be a nonce (number used once) or an alphanumeric sequence. Insome implementations, the initialization vector can be an input similarto a key used with the encryption process. In some implementations, notall encryption mechanisms use an initialization vector. The property ofthe initialization vector can depend on the encryption process. Forexample, for some encryption processes the initialization vector can berandom or pseudorandom. For another example, in other encryptionprocesses the initialization vector is non-repeating. For someencryption schemes, however, the initialization vector can be madepublic even if it is random or non-repeating.

As used herein, a module can be, for example, any assembly and/or set ofoperatively-coupled electrical components, and can include, for example,hardware like a memory, a processor, software (executing in hardware),and/or the like. As used herein, the singular forms “a,” “an” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, the term “a zero-trust storage” is intended to mean asingle zero-trust storage or multiple zero-trust storages with similarfunctionalities. For another example, the term “a data file” is intendedto mean a single data file or multiple data files. Additional detailsregarding zero-trust storage can be found in U.S. patent applicationSer. No. 15/259,863, which has been incorporated herein by reference inits entirety.

FIG. 1 is a schematic block diagram of an index searching system 100,according to an embodiment. In some embodiments, the index searchingsystem 100 includes a client device(s) 103, an index server(s) 109, anda zero-trust storage 101. The zero-trust storage 101 can be operativelycoupled to the client device 103 and the index server 109. Thezero-trust storage 101 includes a security device 105 and a backendstorage 107 operatively coupled to the security device 105.

In some embodiments, the security device 105, the backend storage 107,the index server(s) 109, and the client device 103 are stored and/orexecuted on separate compute devices communicatively coupled togethervia a network (e.g., the Internet) (not shown). Such compute devices caninclude processors (e.g., to execute modules, to use application programinterfaces (APIs), and/or the like) and memories (e.g., to store dataand/or instructions for execution by the processors). In otherembodiments, the security device 105, the backend storage 107, the indexserver(s) 109, and the client device 103 can be stored and/or executedon a common compute device.

The client device 103 can be a compute device (or multiple computedevices) having a processor and a memory. For example, the client device103 can be or include a computer, a cell phone, a tablet, an electronicdevice, a critical embedded system, coprocessors, and field-programmablegate arrays (FPGAs), a document management system, a file system,another storage system, or any system that uses a storage system tostore data for its operation(s). The client device 103 can send searchrequests 121 including search queries (having, for example, strings,commands, keywords, texts, numbers) to the zero-trust storage 101 andreceive search results 127 associated with the search queries and a setof documents from the zero-trust storage 101. In some implementations,the client device 103 can support a set or a subset of operationssimilar to read, write, update, append, and delete. In some instances,the client device 103 can communicate with the security device 105 via anetwork (e.g., the Internet, an intranet, a local area network (LAN), avirtual private network (VPN), a wireless LAN (WLAN), a wired network, awireless network and/or the like). In some implementations, the clientdevice 103 can be a cloud storage system.

The index server(s) 109 can be a compute device (or multiple computedevices) having processing capabilities. For example, in some instances,the index server(s) 109 can be a server such as, for example, a webserver, an application server, a proxy server, a telnet server, a filetransfer protocol (FTP) server, a mail server, a list server, acollaboration server and/or the like. In other instances, the indexserver(s) 109 can be a personal computing device such as a desktopcomputer, a laptop computer, a personal digital assistant (PDA), astandard mobile telephone, a tablet personal computer (PC), and/or soforth. In some implementations, the index server(s) 109 can send a setof index files 129 to zero-trust storage 101 for encryption and storage.In some implementations, the zero-trust storage 101 can retrieve a setof index files 129 from the index server(s) 109. In someimplementations, the index server 109 can be a third-party serverexternal to the zero-trust storage 101 and the client device 103. Insome implementations, the index server 109 is a component of thezero-trust storage 101 and the zero-trust storage 101 can be configuredto generate a set of index files based on a set of documents stored atindex server 109 and/or backend storage 107. In some implementations,the index server 109 is a component of the client device 103 and theclient device 103 can be configured to generate a set of index filesbased on a set of documents stored at index server 109 and/or backendstorage 107.

The backend storage 107 can be configured to store documents and/orother forms of data. The backend storage 107 can be, for example, adevice(s) to implement a cloud storage service such as, for example,Amazon Web Services (AWS)™, Microsoft Azure™, or a physical storagearray, or can be an in-memory database, file system, or a datastructure. An example of a file that can be securely stored in thebackend storage 107 using the security device 105 includes an encryptedsearch index (e.g., a secure index of encrypted data that can be used tosearch the encrypted data) of one or more encrypted documents (stored inthe backend storage 107 and/or other storage locations). For otherexamples, any other suitable file and/or data can be stored in thebackend storage 107. In some implementations, the backend storage 107can support two operations equivalent to download 123 and upload 125(such as receive/send or read/write). The security device 105 cantranslate the upload and download operations to the operations supportedby the backend storage 107. The backend storage 107 can use an internalunit of storage. Some common terms used by the backend storage 107 foreach of the units can be, for example, a bucket, an object, a slot, arecord, a blob, a box, and/or a file.

The security device 105 can include a processor and a memory and can bean interface between the backend storage 107 and the client device 103.The security device 105 can include hardware modules and/or softwaremodules (e.g., stored in memory and/or executed by a processor). Thesecurity device 105 can include any combination of hardware-basedmodules (e.g., a field-programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a digital signal processor (DSP))and/or software-based modules (a module of computer code stored inmemory and/or executed at the processor) capable of performing one ormore specific functions associated with that module. In someembodiments, the security device 105 can be a compute device, a server,a host device, and/or the like. Details of the security device 105 arediscussed herein with regards to FIG. 2.

The security device 105 includes a storage interface such as a blockstorage interface, an object store interface, a remote file system, amountable file system interface and/or the like. In some instances, theclient device 103 does not receive and/or store identifiers associatedwith the location at which the data is stored in the zero-trust storage101 but can refer to the documents using the identifier to which theclient device 103 is accustomed (e.g., using an identifier specific tothe client device 103). For example, the client device 103 can use afile-system interface such as files and directory to refer to eachdocument using a path of the document. The client device 103 can alsorefer to the documents using some kind of document identifier (e.g.,using an identifier specific to the client device 103) as if using anobject store. The client device 103 can thus call the security device105 with the typical file operation such as reading a file or writing afile. The client device 103 can also send a search request 121 to thesecurity device 105 when searching for documents, as described infurther detail herein. In some instances and as described in furtherdetail herein, the security device 105 can be configured to translatesome client device operations into a download request 123 and/or anupload request 125.

In some instances, the zero-trust storage 101 can interact with usersthrough the client device 103. In some instances, the client device 103can, for example, interact with the zero-trust storage 101 using typicalstorage operations (or a subset of) such as read, write, update, append,and delete. The security device 105 can translate the client devicestorage units to the backend storage units and the operations to a setof download and upload operations to be executed by the backend storage107. In some embodiments, the backend storage 107 can be separate from(or not part of) the zero-trust storage 101 because zero-trust storage101 can use any suitable backend storage technology. From theperspective of the client device 103, the backend storage 107 is part ofthe zero-trust storage 101 because the security device 105 overlays andtransforms the backend storage 107 into a zero-trust storage 101.

In some implementations, multiple client devices 103 can be operativelycoupled to the security device 105. Thus, data from these multipleclient devices can be stored to the same backend storage 107 via thesecurity device 105. In some implementations, multiple security devices105 and/or multiple backend storages 107 can be operatively coupled tothe client device 103. Data from the client device 103 can be stored tothe multiple backend storages via the multiple security devices 105.

In use, the security device 105 can receive a set of index files 129from the index server 109 or from the client device 103. In otherimplementations, the security device 105 can receive a set of documentsfrom the index server 109 or from the client device 103 and parse theset of documents to generate a set of index files based on the set ofdocuments. The security device 105 can generate an index identifier foreach index file from the set of index files.

In some implementations, the security device 105 can generate a seedbased at least in part on the index identifier for an index file andcalculate a set of pseudorandom logical block identifiers based on theseed. The set of pseudorandom logical block identifiers is associatedwith a set of storage locations of a database (for example, in thebackend storage 107). In some implementations, the security device 105can calculate a set of pseudorandom logical block identifiers directlybased on the index identifier without generating a seed. In someimplementations, the security device 105 can generate a seed, based atleast in part on the index identifier, using a cryptographicpseudorandom hash function with the index identifier as an input to thecryptographic pseudorandom hash function. In some implementations, thesecurity device 105 can use a cryptographic key (e.g., as an input tothe cryptographic pseudorandom hash function) in addition to the indexidentifier to calculate the set of pseudorandom logical blockidentifiers.

The security device 105 can parse the index file into a set of indexdata portions and store (or write, or upload) each index data portionfrom the set of index data portions at a different storage location inthe backend storage 107 as indicated by a different identifier from theset of pseudorandom logical block identifiers. In some implementations,the security device 105 can parse the index file by the set of indexentries. In such implementations, each index data portion can includeone index entry defining an association of a characteristic (e.g., aword, a letter, a phrase, or meta-data) in a set of documents with a setof information (e.g., each document identifier for each document thatcontains the characteristic) associated with the set of documents. Forexample, an index file includes two index entries, (1) the first indexentry defining an association of phrase “San Francisco” with a first setof document identifiers for a first set of documents that contain phrase“San Francisco”, and (2) the second index entry defining an associationof phrase “software engineers” with a second set of document identifiersfor a second set of documents that contain phrase “software engineers.”The security device 105 can parse the index file into two index dataportions, the first index data portion including the first index entryand the second index data portion including the second index entry.

In some implementations and as described in further detail herein, eachindex data portion, which includes one index entry, can be furtherparsed (e.g., randomly) into a set of secondary index data portions.Each secondary index data portion from the set of secondary index dataportions can be stored at a different storage location associated with apseudorandom logical block identifier.

The security device 105 can store each index data portion by sending asignal to the backend storage 107 or to a database in the backendstorage 107. In some implementations, the set of index data portions canbe stored at different storage systems, different storage locationswithin the same storage system (e.g., the backend storage 107), ordifferent databases within the same storage system.

In some implementations, the security device 105 can encrypt each indexdata portion from the set of index data portions prior to storing eachindex data portion in the backend storage 107 or prior to sending asignal to the database to store each index data portion in the backendstorage 107. In some instances, in response to calculating the set ofpseudorandom logical block identifiers and prior to storing each indexdata portion in the backend storage 107, the security device 105 canretrieve data stored at each storage location within the backend storage107 indicated by each identifier from the set of pseudorandom logicalblock identifiers. The advantage of retrieving data stored at eachstorage location associated with each pseudorandom logical blockidentifier and then storing each index data portion at each storagelocation is to hide the operations within the zero-trust storage 101from other devices (external to zero-trust storage 101) such that theoperations are unknown to the other external devices for security andprivacy purposes.

In some implementations, prior to storing each index data portion ateach storage location in the backend storage 107, the security device105 can determine that each storage location associated with eachpseudorandom logical block identifier does not have existing datastored. For example, the security device 105 can ensure that writingeach index data portion at each storage location in the backend storage107 will not overwrite existing data stored in that storage location. Insome instances, for example, the security device 105 can ensure that anidentifier stored with data at each storage location indicates that datacan be written at that storage location (e.g., upon initializationand/or deletion of data, an identifier can be modified to indicate thestorage location is available to be written). In some instances, if suchan identifier instead indicates that other data is stored at thatlocation, the security device 105 does not write data to that locationand selects another storage location identified by another pseudorandomlogical block identifier at which to store data.

An index file can associate a characteristic (e.g., a word, a letter, aphrase, or meta-data) in a set of documents with a set of informationassociated with the characteristic in the set of documents. The set ofinformation can include, for example, a document identifier (1) from aset of document identifiers and (2) for each document from the set ofdocuments, a set of locations of the characteristic in the set ofdocuments, a set of statistical information associated with thecharacteristic in the set of documents, a relationship informationassociated with a relationship between a first characteristic in the setof documents and a second characteristic in the set of documents, and/orthe like. For example, the index file can include associations of phrase“San Francisco” with (1) the document identifier of each document thathas the phrase “San Francisco”, (2) the location in a storage (e.g.,backend storage 107) of each document that has the phrase, (3) thelocation of the phrase in each document (e.g., page 2, paragraph 3, line4), (4) how many times the phrase appears in each document, (5) theother words that are in the same sentence of the phrase in eachdocument, (6) the document identifier of each document that has thephrase “San Francisco” and the phrase “software engineer,” and/or thelike.

In use, the security device 105 can receive a search request 121including a search string from the client device 103. The search stringcan be associated with a characteristic (e.g., a phrase, a word, aletter, or meta-data) in a set of documents within a database in thebackend storage 107. The security device 105 can identify, based on thesearch string, a set of logical block identifiers associated with a setof storage locations of the database and retrieve index data stored ateach storage location from the set of storage locations using the set oflogical block identifiers. In some implementations, the set of logicalblock identifiers is a set of pseudorandom logical block identifiers. Insome implementations, the security device 105 can identify a seed basedat least in part on the search string, and can calculate the set ofpseudorandom logical block identifiers for the database based on theseed, as described above. In other implementations, the security device105 can identify the set of logical block identifiers using acryptographic pseudorandom hash function with the seed as an input tothe cryptographic pseudorandom hash function. In such implementations,the seed can be at least a portion of the search string.

In some embodiments, the security device 105 (e.g., by the randomizermodule 213 in FIG. 2) can determine if the retrieved index data storedat each storage location includes a hash value produced by the documentmeta-data and/or an identifier associated with the document.Specifically, the security device 105 can determine whether theretrieved index data are part of the requested document by matching oneor more identifier(s) stored with the index data (or a hash valueproduced based on the identifier(s)) and the hash value (or otheridentifier). If none of the retrieved index data match the hash value,the document is determined to not be stored in the backend storage 107.If at least one block of index data matches the hash value, anyadditional blocks of index data can be retrieved (e.g., by the storagemodule 219 in FIG. 2) using remaining numbers from the sequence ofnumbers. After the blocks of index data associated with the searchrequest have been retrieved and decrypted, they can be pieced togetherand/or compiled (e.g., using the sequence of numbers to identify order).The compiled document (i.e., the search result file) can then bereturned to the client device 103. In some instances, the search resultfile can be a portion of the index file originally stored. In someinstances, data stored at locations identified by the set of logicalblock identifiers that does not include the identifier associated withthe requested document, can be ignored by security device 105. In otherinstances, data stored at locations identified by the set of logicalblock identifiers that does not include the identifier associated withthe requested document can still be read and rewritten to the backendstorage 107 (e.g., via instructions from security device 105) to obscurefrom the backend storage 107 which storage locations are associated withthe requested document. Such data, however, is not returned to theclient device 103.

The security device 105 can then compile a search result file based onthe index data stored at each storage location from the set of storagelocations and identify, based on the search result file, a set ofinformation associated with the characteristic in the set of documents.The set of information can include a set of document identifiers. Eachdocument identifier from the set of document identifiers is associatedwith a document from the set of documents associated with thecharacteristic.

For example, the security device 105 receives a search request 121,which requests to retrieve each document identifier associated with adocument that includes phrase “software engineer.” In instances wheneach index entry is stored at a single storage location, the securitydevice 105 can identify, based on the phrase “software engineer”, alogical block identifier associated with a storage location (e.g.,within the backend storage 107) and retrieve the index entry stored atthe storage location. The security device can then send a signal to theclient device 103 that includes each document identifier associated witha document that includes the phrase “software engineer” (e.g., asidentified by the index entry). In instances when the index entryassociated with phrase “software engineer” is parsed into a set ofsecondary index data portions, the security device 105 can identify,based on the phrase “software engineer”, a set of logical blockidentifiers associated with a set of storage locations of the databaseand retrieve each secondary index data portion stored at each storagelocation from the set of storage locations using the set of logicalblock identifiers. In such instances, the security device 105 can thencompile a search result file (e.g., including the index entry orentries) based on the set of secondary index data portions and identify,based on the search result file, each document identifier associatedwith a document that includes phrase “software engineer.”

For another example, the security device 105 receives a search request121, which requests to retrieve each document identifier associated witha document that includes phrase “San Francisco” and phrase “softwareengineer.” In some instances, the security device 105 can identify,based on phrases “San Francisco” and “software engineer”, a set oflogical block identifiers associated with a set of storage locations ofthe database. When a set of logical block identifiers are identifiedbased on the combined search string “San Francisco” and “softwareengineer”), the security device 105 can retrieve index data (e.g.,representing index entries) stored at each storage location from the setof storage locations using the set of logical block identifiers. Theindex entries based on the index data can include identifiers ofdocuments that include both “San Francisco” and “software engineer”.

When a set of logical block identifiers are not identified, the securitydevice 105 can identify, based on phrase “San Francisco”, a first set oflogical block identifiers associated with a first set of storagelocations of the database, and identify, based on phrase “softwareengineer”, a second set of logical block identifiers associated with asecond set of storage locations of the database. The security device 105can retrieve first index data (e.g., representing one or more indexentries representing “San Francisco”) stored at each storage locationfrom the first set of storage locations and second index data (e.g.,representing one or more index entries representing “software engineer”)stored at the second set of storage locations. The security device 105can compare the first index data and the second index data and generatea search result file that includes only overlapping document identifiers(e.g., index entries from both the first index data and the second indexdata). The overlapping document identifiers are associated withdocuments that includes both phrase “San Francisco” and phrase “softwareengineer.”

In some implementations, the index data can be stored at differentstorage systems, different storage locations within the same storagesystem (e.g., the backend storage 107), or different databases withinthe same storage system. When the index data are stored at differentdatabases within the same storage system, the set of logical blockidentifiers is associated with the set of storage locations of thedifferent databases. The security device 105 can send a signal thatincludes the set of information to the client device 103.

In some implementations, in addition to encrypting index files (or indexdata portions) associated with a set of documents, the security device105 (e.g., the encryption module 215 in FIG. 2) can encrypt the set ofdocuments and/or each portion of a set of portions of each document fromthe set of documents. Once a client device selects a document identifierfrom the search result file, the security device 105 can retrieve anddecrypt each encrypted portion of the document associated with thedocument identifier. The security device 105 can subsequently compilethe document based on the decrypted portions of the document and sendthe document to the client device 103. The zero-trust storage 101 can beconfigured to provide secure index search within a set of documents andprovide decrypted document.

The security device 105 can modify an initialization vector for theindex data stored at each storage location from the set of storagelocations to define modified index data for each storage location fromthe set of storage locations. In some implementations, the securitydevice 105 can encrypt the modified index data for each storage locationfrom the set of storage locations prior to sending a signal to thedatabase to write to the set of storage locations. The security device105 can send the signal to the database to write the modified index datafor each storage location from the set of storage locations to thatstorage location from the set of storage locations within the database.

FIG. 2 is a schematic block diagram of a security device 205 within azero-trust storage, according to an embodiment. The security device 205within a zero-trust storage (such as the zero-trust storage 101 inFIG. 1) can include a security module 206 (also referred to herein as asecurity controller), an index module 230 (also referred to herein as anindexer), a processor 220, and a memory 210. The security module 206 caninclude an interface module 211 (also referred to herein as an interfacecontroller), a randomizer module 213 (also referred to herein as arandomizer), an encryption module 215 (also referred to herein as anencryptor), a decryption module 217 (also referred to herein as andecryptor), and a storage module 219 (also referred to herein as astorage controller). The security device 205 can be structurally andfunctionally similar to the security device 105 discussed with respectto FIG. 1.

Each operation(s) of the interface module 211, the randomizer module213, the encryption module 215, the decryption module 217, the storagemodule 219, the index module 230, as well as the memory 210, can becontrolled and/or executed by the processor 220. Each module orcomponent in the security device 205 can be hardware modules and/orsoftware modules (stored in memory 210 and/or executed by a processor220). Each module or component in the security device 205 can be anycombination of hardware-based modules (e.g., a field-programmable gatearray (FPGA), an application specific integrated circuit (ASIC), adigital signal processor (DSP)) and/or software-based modules (a moduleof computer code stored in memory 210 and/or executed at the processor220) capable of performing one or more specific functions associatedwith that module. Each module or component in the security device 205can be operatively coupled to the other modules/components in thesecurity device 205. As discussed herein, the security device 205 andits components can be stored and/or executed on a server, host device,client device, and/or the like. In some embodiments, some components ofthe security device 205 can be stored and/or executed on a first device(e.g., a server) and other components of the security device 205 can bestored and/or executed on a second device (e.g., a client device)operatively coupled to the first device via a network (e.g., theInternet).

In some instances, the interface module 211 is operatively coupled witha client device (such as the client device 103 in FIG. 1) and therandomizer module 213. The randomizer module 213 is operatively coupledwith the storage module 219, the encryption module 215, and thedecryption module 217. Because the communication with the storage module219 can be via encrypted logical blocks of data, the randomizer module213 can use the encryption module 215 to encrypt logical data blocksbefore sending the encrypted logical data blocks to the storage module219. The randomizer module 213 can also use the decryption module 217 todecrypt logical data blocks after receiving the data blocks from thestorage module 219.

The interface module 211 is a translation layer of the security device205 that interfaces with the client device (such as the client device103 in FIG. 1). Depending on the client device, the interface module 211can support different client device storage application programminginterfaces (APIs). Functionality-wise, the interface module 211 cansupport search, read, write, update, append, delete requests and/orother suitable instructions. In some cases, the interface module 211 canreceive a request (e.g., a search request) from the client device andcan then translate the request to a uniform request and/or a requestused and recognized by the components of the security module 206. Theinterface module 211 can also translate an identifier used by the clientdevice (also referred herein to as client identifier) to a storagesystem identifier used by the rest of the system (e.g., the remainingmodules of the security module 206 and/or the backend storage), and aseed used by the randomizer module 213 described in further detailherein. In some instances, the client identifier (e.g., a search string)can be translated and/or used to define a seed using a seed mappingfunction (e.g., a hash function such as a cryptographic pseudorandomhash function) with the client identifier as an input to the seedmapping function. The seed mapping function can then be used as thetranslation function. In other instances, any function that translatesan identifier to a number (or a set of numbers) deterministically can beused by a seed mapping function. Using a deterministic function for theseed-generating function optimizes and/or improves an accessing patternof the backend storage. The more random the locations of the actualstorage units used by backend storage, the more difficult it can be forthe backend storage (or anyone who has access to the backend storagesuch as a system administrator of the system) to figure out and/oridentify the potential locations of the storage units used by thebackend storage to store a specific file or datum. Thus, data (e.g.,index files) can be privately and randomly stored at the backendstorage.

The randomizer module 213 can be configured to translate the uniformaccess request from the interface module 211 to a sequence of downloadand upload requests. In some instances, the input from the interfacemodule 211 includes an identifier used by the remaining modules of thesecurity module 206 to refer to the document, the index file, the seed,the operation, and/or the document content in the case of write/append.In some instances, both a download request and an upload request areexecuted to ensure the backend storage cannot differentiate theoperations among read, write, update, append, and delete. Similarlystated, each of the requests look the same to the backend storagesystem. This prevents leaking an access pattern of zero-trust storage(such as the pattern used by the zero-trust storage 101 in FIG. 1).

The randomizer module 213 can use the seed as an input to a pseudorandomnumber generator to identify (or generate or calculate) a set ofpseudorandom numbers. The randomizer module 213 can then map the set ofpseudorandom numbers to a set of logical block numbers (or a set oflogical block identifiers) that are used by the storage module 219 tostore data at storage locations (e.g., in the backend storage 107 ofFIG. 1) associated with the set of logical block numbers. In someimplementations, the set of logical block identifiers are not stored inlong-term memory (e.g., are not stored in persistent memory at securitydevice 205 or in the backend storage 107 of FIG. 1). Instead, each timethe client device accesses specific data or queries based on specificcharacteristics, the set of logical block identifiers can be identifiedand/or calculated using the pseudorandom number generator and the seed.

Each logical block identifier can map to the storage unit used by theunderlying backend storage as described below. In this implementation,the seed can be used to map to the location of blocks of data (e.g.,index files) stored in the backend storage. Similarly stated, the seedcan be used as a key to identify where to store the blocks of data(e.g., index files) in the backend storage and similarly which blocks ofdata (e.g., index files) to retrieve from the backend storage. In otherimplementations, the seed can be combined with an additionalcryptographic key (or other identifier) to generate a sequence ofnumbers identifying where to store the blocks of data (e.g., indexfiles) in the backend storage in a write operation and, similarly, whichblocks of data (e.g., index files) to retrieve from the backend storagein a read operation. Similarly stated, using the seed and thecryptographic key, the sequence generated by the randomizer module 213can be further randomized. Each number from the sequence of numbers canmap to a location in the backend storage storing a portion of the data(e.g., index data portions or index data). The portion of data stored ateach memory block of the backend storage pertains to the data file(e.g., index file).

In some implementations, the identifier used by the client device (theclient identifier, e.g., a search string) for a query, a specific fileand/or access request can be processed by the randomizer module 213 andmapped to a seed mapping function (also referred to herein as a “mappingfunction”). The seed mapping function can be, for example, a hashfunction such as a cryptographic pseudorandom hash function. In such aninstance, each time the client device requests a query, a specific fileor data, the randomizer module 213 can use the seed mapping function(with the identifier used by the client device as an input) to identify,generate and/or calculate the seed. Thus, in such instances, therandomizer module 213 can be configured to calculate the seed each timethe client device requests data from the zero-trust storage. In suchinstances, the randomizer module 213 can use the client identifier(e.g., a search string) for a particular query, a file and/or block ofdata as an input to the seed mapping function to recalculate and/orrecreate the seed each time a request is made with that clientidentifier. The sequence of numbers (the set of logical blockidentifiers) generated by the seed can then be used to retrieve theblocks of data (e.g., index data portions) associated with that fileand/or data. The set of logical block identifiers is associated with aset of storage locations of a database in the backend storage. In otherinstances, a mapping between the client identifier and the seed can beused to identify the seed. In still other instances, the clientidentifier can be the seed used to identify the sequence of numbers (theset of logical block identifiers) used to store and/or retrieve thedifferent blocks of data associated with that file and/or data. In someinstances, the client identifier is defined by and is specific to theclient device (e.g., a file path used by the client device to store andaccess the file and/or data).

In some instances, the seed generation can be based on the clientidentifier (e.g., a search string) or a uniquely identifying property ofthe data such as a file name, a file path, a record locator, a hash ofone or more properties of the data, and/or the like. This allows a userof the system to regenerate the seed from the same client identifier(e.g., a search string) or uniquely identifying property of the data.Given the same seed mapping function, the same pseudorandom numbergenerator process, and the same set of cryptographic keys, the securitymodule 206 can uniquely map to the correct possible set of data blocksused by the backend storage.

The encryption module 215 can be used by the randomizer module 213 toencrypt each data block (e.g., portion of a file, an index data portion)before uploading the data block to the backend storage through thestorage module 219. The encryption scheme used by the encryption module215 can be any suitable encryption scheme. In some implementations, theencryption scheme can be modified and/or pluggable as long as theencryption can be decrypted (e.g., by the decryption module 217) and canscale to the logical block size (e.g., the size of each portion of thefile) used by the security module 206. This allows the overall system toswap out “old” encryption schemes that are later found to be either tooslow or unsecure and to replace such old encryption schemes with new andimproved encryption scheme(s). In some implementations, the encryptionmodule 215 and the decryption module 217 have access to the encryptionkey, but the interface module 211, the randomizer module 213 and thestorage module 219 do not have access to the encryption key. Thus, insome implementations, the encryption module 215 and/or the decryptionmodule 217 can also interact with a key management system (not shown inFIG. 2) and/or a trust platform module (not shown in FIG. 2). In otherimplementations, the encryption key can be implemented and/or stored byan existing key management system (not shown) either in hardware orsoftware (stored in memory and/or executed in a processor). This makesthe architecture extremely flexible and secure because key use isisolated to a small portion of the system and the key is not propagatedto other modules of the security module 206 or the backend storagesystem. In such implementations, this helps to ensure that the backendstorage is zero-trust. In still other implementations, the seedgenerated by the randomizer module 213 can be used as an input to theencryption module 215 and used as an input and/or an encryption key toan encryption function to encrypt the data. In such implementations, forexample, the final encryption key can be a function of the storedencryption key and the seed. In such a manner, the final encryption keycan be further obscured based on the seed.

The decryption module 217 can be configured to receive and decryptencrypted data (e.g., encrypted index data portion) stored in thebackend storage using a decryption key that can be a compliment to theencryption key. Such a decryption key can be the same as the encryptionkey, can be stored with the encryption key, can be stored remote fromthe encryption key, can be paired with the encryption key, and/or thelike. In other implementations, the seed can be used by the decryptionmodule 217 as a decryption key and/or as a way to further obscure thedecryption key when decrypting the data. In other instances, the samemodule can act as both the encryption and the decryption modules.

The storage module 219 can be configured to interface with a backendstorage. The storage module 219 can handle reading/writing to theunderlying storage system using an API or abstraction exposed by thebackend storage and mapping the logical block used by the randomizermodule 213 to the storage unit used by the backend storage. For example,if AWS S3 is used, the storage module 219 API will be using the S3 APIto upload and download encrypted storage unit to the S3 system. Thestorage module 219 can translate the logical blocks to the S3 storageunits such as buckets and/or objects. The storage module 219 can beconfigured to hide and/or obscure the different interfaces (APIs) of theunderlying backend storage so the zero-trust security module 206 cantreat the backend storage as a “dumb” storage system with only upload ordownload operations. This allows the zero-trust storage (such as thezero-trust storage 101 in FIG. 1) to be backend agnostic and pluggable.Accordingly, different backend storage and/or protocols can be used.This also allows the zero-trust storage to secure either a cloud storagesystem or a physical storage array. In some instances, data passingthrough the storage module 219 is encrypted by the encryption module 215prior to being sent to the storage module 219.

The index module 230 can be configured to parse components (e.g., words,letters, phrases, or meta-data) of a set of documents or other form ofmedia and insert into forward or inverted indices to generate a set ofindex files. Such document parsing (similar to search engine indexing)can involve natural language processing. In some implementations, thesecurity device 205 can receive a set of index files from index serversor from a client device. The index module 230 can generate an indexidentifier for each index file of the set of index files.

The memory 210 can be, for example, a random-access memory (RAM) (e.g.,a dynamic RAM, a static RAM), a flash memory, a removable memory, a harddrive, a database and/or so forth. In some implementations, the memory210 can include (or store), for example, a database, process,application, virtual machine, and/or other software modules (storedand/or executing in hardware) and/or hardware modules configured toexecute an index searching process and/or one or more associated methodsfor the zero-trust storage. In such implementations, instructions forexecuting the index searching process and/or the associated methods canbe stored within the memory 210 and executed at the processor 220.

The processor 220 can be configured to, for example, write data into andread data from the memory 210, and execute the instructions storedwithin the memory 210. The processor 220 can also be configured toexecute and/or control, for example, the operations of the securitymodule 206, including the interface module 211, the randomizer module213, the encryption module 215, the decryption module 217, the indexmodule 230, and the storage module 219. Similarly stated, the functionsand/or operations described with respect to security module 206 (andassociated modules) can be implemented and/or executed by processor 220.In some implementations, based on the methods or processes stored withinthe memory 210, the processor 220 can be configured to execute indexsearching processes, as described in FIGS. 3-5.

While shown and described in FIG. 2 as being within a single securitydevice 205, in other embodiments, the components and modules of thesecurity device 205 can be distributed between multiple devices. Forexample, in such other embodiments, the interface module, randomizermodule, encryption module, decryption module, the index module, and/orstorage module can be distributed across two or more compute devicesoperatively coupled via a network. In such other embodiments, the one ormore compute devices can work together to implement the functions of thesecurity module.

FIG. 3 is a flowchart illustrating an initialization process 300,according to an embodiment. This initialization process 300 can beimplemented at a processor and/or a memory (e.g., processor 220 ormemory 210 as discussed with respect to FIG. 2) of a security device(such as the security device 105 in FIG. 1 and the security device 205in FIG. 2). In some instances, the initialization process 300 can beused to prepare a backend storage (such as the backend storage 107 inFIG. 1) for implementing zero-trust storage (such as the zero-truststorage 101 in FIG. 1). In some instances, the initialization process300 can be executed before any documents (e.g., index files) areuploaded or as part of a batch upload operation for an initial set ofdocuments (e.g., index files). In other instances, the steps in theinitialization process 300 can be performed as each memory block (e.g.,storage location) is used for the first time (and not for subsequentuse). By performing the steps of the initialization process as astandalone step or during an initial batch upload operation, theinitialization process 300 can effectively hide existence of documents(e.g., index files) in the zero-trust storage.

At 302, the processor (such as the processor 220 of the security device205 in FIG. 2) determines a block size of the backend storage (e.g., ablock size supported by the backend storage). At 304, the processor thenbreaks the backend storage into a set of logical memory blocks based onthe block size. To break the backend storage logical storage unit intothe logical memory blocks, the security device can use a logical blocksize. This logical block size can be determined either algorithmicallybased upon sample documents or as an input specified by the user (or asystem administrator). Each logical memory block from the set of logicalmemory blocks is associated with a different logical memory blockidentifier (to be used by the processor to reference storage locationswithin a backend storage). To treat the backend storage uniformly, thesecurity device (e.g., the randomizer module and/or the storage moduleexecuted by a processor) can treat the backend storage as a logicalstorage with a naming scheme based on the logical memory blockidentifiers. The security device (e.g., the randomizer module and/or thestorage module executed by a processor) can, according to the actualbackend storage used, map the logical memory block identifiers to eachstorage unit and/or identifier used by the backend storage. As such, thesecurity device can translate and/or map the logical memory blockidentifiers as used by the security device to the identification schemeused by the backend storage.

At 306, the security device (e.g., the randomizer and/or storage moduleexecuted by a processor) can differentiate logical memory blocks markedas free and logical memory blocks marked as used by documents already inthe security device. One way to do so is by using a pre-determined hashor identifier value (e.g., all zeroes, all ones) or a set of hash oridentifier values to identify free logical memory blocks. Alternatively,a function can be applied that determines whether a logical memory blockor a set of multiple logical memory blocks are free. In either case, foreach logical memory block, at 308, the security device can implement (1)a process to determine whether a logical memory block in the backendstorage is available (and/or free) or unavailable, and/or (2) a processto mark a logical memory block as available (and/or free) orunavailable, at 310 (e.g., by writing the hash value to the memory toidentify a free or available block).

In some instances, at 312, an initialization vector for each logicalmemory block can be determined and/or identified (e.g., by theencryption module). In some instances, for example, for encryptionprocesses in which the initialization vectors are non-repeating (but notrandom), a counter can be used for the initialization vector. Forexample, an initial initialization vector can have a value of all zeroesand can be incremented each time a new value is used. For anotherexample, an initial initialization vector can use all ones and can bedecremented each time a new value is used. In some instances, the sameinitialization vector can be used for each block of memory beinginitialized. Alternatively, an initialization vector can be randomlygenerated for each block of memory.

In some instances, an initial content of each logical memory block canbe pre-determined or generated randomly, at 314. Such a pre-determinedvalue can be a value (such as all zeroes or all ones) or the content ofdocuments being batch uploaded as part of initialization. Regardless ofblock content, because encryption randomizes the output, afterencryption, the encrypted logical memory blocks can appearcryptographically non-distinguishable.

In some instances, at 316, the processor encrypts each logical memoryblock (i.e., the data within that logical memory block) using theinitialization vector and the encryption key used for the zero-truststorage. The processor then uploads and writes the logical memory blocksto the backend storage, at 318. Because each logical memory block isencrypted and then uploaded to the backend storage and because theencryption key is not uploaded to the backend storage, the backendstorage is unable to determine whether any documents (e.g., index files)are actually being uploaded along with the initialization step.

FIG. 4 is a flowchart illustrating an index searching process 400,according to an embodiment. This index searching process 400 can beimplemented at a processor and/or a memory (e.g., processor 220 ormemory 210 as discussed in FIG. 2) of a security device (such as thesecurity device 105 in FIG. 1 and the security device 205 in FIG. 2). Asearch request including a search string (e.g., a phrase, a word, aletter, a number, a symbol, and/or the like) is received from a computedevice, at 401. The search string is associated with a characteristic(e.g., a phrase, a word, a letter, or meta-data) in a set of documentswithin a database (e.g., stored in a backend storage). In someimplementations, the search string can be converted to a hash value(e.g., using any suitable hash function). This hash value can be used asan input to a seed mapping function to generate a seed. In otherinstances, the hash value can be used as a seed. In still otherinstances, the search string can be used as a seed. In still otherinstances, any other suitable identifier can be used instead of a hash.

The search string (and/or any other suitable parameters such as a range,a seed a cryptographic key and/or the like) can be used as an input(e.g., at a pseudorandom number generator) to generate a sequence ofnumbers (e.g., using any suitable sequence generating function or“pseudorandom number generator”) (also referred to herein as a set oflogical block identifiers), at 402. This sequence of numbers can then beused (e.g., by the storage module) to retrieve data (e.g., index data)from a set of memory blocks (e.g., retrieve data at a set of storagelocations) from the backend storage, at 403. As discussed above, using apseudorandom number generator to produce the sequence of numbers thatcorrespond to the set of memory blocks randomizes where in the backendstorage each block of data is stored. Similarly stated, the sequence ofnumbers can correspond to and/or be associated with memory locations inthe backend storage.

The set of data blocks (i.e., index data) can then be sent (e.g., by therandomizer module) to the decryption module, which can decrypt the datablocks (i.e., index data) and return decrypted data blocks (to therandomizer module, for example). In other instances, the data in thebackend storage is not encrypted, and thus, the data blocks (i.e., indexdata) are not decrypted (e.g., by the decryption module).

In some instances, it can be determined (e.g., by the randomizer module213 in FIG. 2) if the retrieved index data stored at each storagelocation include a hash value produced by the document meta-data and/oran identifier associated with the document. Specifically, it can bedetermined whether the retrieved index data are part of the requesteddocument by matching one or more identifier(s) stored with the indexdata (or a hash value produced based on the identifier(s)) and the hashvalue (or other identifier). If none of the retrieved index data matchthe hash value, the document is determined to not be stored in thebackend storage. If at least one block of index data matches the hashvalue, any additional blocks of index data can be retrieved (e.g., bythe storage module 219 in FIG. 2) using remaining numbers from thesequence of numbers. After the blocks of index data associated with thesearch request have been retrieved and decrypted, they can be piecedtogether and/or compiled (e.g., using the sequence of numbers toidentify order) to produce a search result file, at 404. In someinstances, the compiled document (i.e., the search result file) can thenbe returned to the client device 103.

The search result file (which can be a portion of the uploaded indexfile) can associate a characteristic (e.g., a word, a letter, a phrase,or meta-data) in a set of documents with a set of information associatedwith the characteristic in the set of documents. The set of informationcan include, for example, a document identifier (1) from a set ofdocument identifiers and (2) for each document from the set ofdocuments, a set of locations of the characteristic in the set ofdocuments, a set of statistical information associated with thecharacteristic in the set of documents, a relationship informationassociated with a relationship between a first characteristic in the setof documents and a second characteristic in the set of documents, and/orthe like. For example, the search result file can include associationsof phrase “San Francisco” with (1) the document identifier of eachdocument that has the phrase “San Francisco”, (2) the location in thestorage (e.g., backend storage 107) of each document that has thephrase, (3) the location of the phrase in each document (e.g., page 2,paragraph 3, line 4), (4) how many times the phrase appears in eachdocument, (5) all the other words that are in the same sentence of thephrase in each document, (6) the document identifier of each documentthat has the phrase “San Francisco” and the phrase “software engineer,”and/or the like. In some instances, each document identifier from theset of document identifiers can be associated with a document from theset of documents associated with the characteristic. In some instances,the set of information is identified based on the search result file andsent as a first signal to the compute device, at 405.

In some implementations, the index data can be stored at differentstorage systems, different storage locations within the same storagesystem (e.g., the backend storage 107), or different databases withinthe same storage system. When the index data are stored at differentdatabases within the same storage system, the set of logical blockidentifiers is associated with the set of storage locations of thedifferent databases.

In some instances, to hide data that may provide enough information ofthe document (or the index file, or the client device) to compromise thedocument and/or the system, the index searching process 400 can beexecuted regardless of operations (i.e., whether the actual operation issearch, read, write, append, delete, or update). For example, even ifthe user knows that a new document (e.g., index file) is being writtenand there is thus no need to check whether this document (e.g., indexfile) already exists in the system, for the overall security of thesystem and to hide from the backend storage that a new document (e.g.,index file) is being written, the index searching process can be run tofetch the data blocks corresponding to the sequence generated from theseed based on the document identifier. In some instances, the indexsearching process can (1) obscure the access pattern of the backendstorage by downloading a set of data blocks with each operation, and (2)download an initial set of data blocks to either identify whether adocument (e.g., index file) exists in the backend storage already (toread/return the content, delete the content, or append/update thecontent), or to find suitable free logical memory blocks for newdocument (write) or new content of the existing document (append/updatethe content).

In some instances, an initialization vector for the index data stored ateach storage location from the set of storage locations is modified todefine modified index data for each storage location from the set ofstorage locations, at 406. The modified index data for each storagelocation from the set of storage locations is encrypted prior to sendinga second signal to the database. Finally, a second signal can be sent tothe database to write the modified index data for each storage locationfrom the set of storage locations to that storage location from the setof storage locations within the database, at 407.

FIG. 5 is a flowchart illustrating an index file upload process 500 inthe case of a read operation, according to an embodiment. The index fileupload process 500 can be implemented at a processor and/or a memory(e.g., processor 220 or memory 210 as discussed in FIG. 2) of a securitydevice (such as the security device 105 in FIG. 1 and the securitydevice 205 in FIG. 2). In some instances, the index file upload process500 can be performed after an index file is received at 501 (e.g., froma third party server, or from a client device). The index fileassociates a characteristic in a set of documents with a set ofinformation associated with the characteristic in the set of documents.In some implementations, a set of documents can be received (e.g., fromthe index server or from the client device) and parsed to generate a setof index files based on the set of documents (e.g., by an index module).An index identifier can be generated for each index file from the set ofindex files, at 502.

In some instances, the index file upload process 500 is performed aftersearch results are identified in response to a search request. Forexample, the method 400 of executing the index searching processincludes, at 406, modifying an initialization vector for the index datastored at each storage location from the set of storage locations todefine modified index data for each storage location from the set ofstorage locations. Subsequently, the method 400 of executing the indexsearching process includes, at 407, sending a second signal to thedatabase to write the modified index data for each storage location.Steps 406 and 407 can be executed based on the index file upload process500 described herein. In some instances, regardless of whether the indexfile already exists in the backend storage, a set of data blocks fromthe index searching operation can be received (e.g., by the randomizermodule). The set of data blocks can correspond to the sequence generatedfrom the pseudorandom number generator based on a seed generated from aseed mapping function (e.g., hashing or other mapping) using thedocument identifier (or the search string) as an input to the seedmapping function.

For example, depending on whether the randomizer module stores the indexidentifier, the hash, the seed, and/or the sequence in memory, therandomizer module may or may not recalculate the hash, regenerate theseed, and/or regenerate the sequence, but these inputs can be used bythe index file upload process 500 as well as the index searchingprocess. In some implementations, when the randomizer module does nothave the seed, the generated sequence, the hash, and/or the set of datablocks, a hash value (e.g., identifier) and/or a seed can beregenerated/recalculated, and a set of pseudorandom logical blockidentifiers (or a sequence) can also be regenerated, at 503. Such newlygenerated set of pseudorandom logical block identifiers, hash value(e.g., a document identifier), seed, and/or the sequence can be storedand used in other steps of an index searching process (as described withrespect to FIG. 4). Such newly generated set of pseudorandom logicalblock identifiers, hash value (e.g., a document identifier), seed,and/or the sequence can be stored temporarily (e.g., in a buffer) onlyfor this operation and removed from memory (e.g., the buffer) after thisoperation finishes, stored temporarily (e.g., in a buffer) for a set ofoperations but removed from memory (e.g., the buffer) after a period oftime, or stored permanently in long-term memory (such as memory 210 inFIG. 2). The set of pseudorandom logical block identifiers is associatedwith a set of storage locations of a database (for example, in thebackend storage 107.) In other implementations, the randomizer modulecan already have the seed, the sequence, the hash, and the set of datablocks downloaded based on the index searching process.

At 504, the index file can be parsed into a set of index data portionsand each index data portion from the set of index data portions can bestored (or written or uploaded) at a different storage location in thebackend storage as indicated by a different identifier from the set ofpseudorandom logical block identifiers. Prior to storing each index dataportion at each storage location, data stored at each storage locationfrom the set of storage locations can be retrieved, at 505, using theset of logical block identifiers for the purpose of obfuscating theindex file upload operation from potential attackers. Subsequently, eachindex data portion can be stored by sending a signal to the backendstorage or to a database in the backend storage, at 506. In someimplementations, the set of index data portions can be stored atdifferent storage systems, different storage locations within the samestorage system (e.g., the backend storage), or different databaseswithin the same storage system.

To obfuscate the index file upload operation from potential attackersthat are able to observe a communication network, a zero-trust storage(e.g., the zero-trust storage 101 in FIG. 1) and/or storage operationsof the backend storage, the index file upload process can be followed orproceeded by a download process (or an index searching process). Inother words, to obfuscate the index file upload (or download) operationfrom potential attackers, both the index file upload process and theindex file download process are performed in each operation. The indexfile upload process can be proceeded or followed by the index filedownload process in such operations. In other instances, thisobfuscation can be eliminated if network efficiency is more importantthan the security from the obfuscation. The purpose of the obfuscationis to prevent attackers from differentiating the upload operation fromthe other operations (delete/write/update/append).

For each data block (i.e., index data portion) associated with the setof pseudorandom logical block identifiers, the index file upload processcan re-encrypt each data block (i.e., index data portion) and thenupload the data blocks (i.e., index data portion) back to the backendstorage. Thus, from the perspective of backend storage, for either aread or write operation, the data blocks (i.e., index data portion)stored within a set of logical memory blocks are downloaded, and theneach data block is uploaded back into the system regardless of whetherthe data block value changed. The same set of operations (download andthen upload to rewrite each logical memory block) is also used in otheroperations (delete/write/update/append).

In some instances, the re-encryption process can include: (1) decryptthe block of data, (2) update the initialization vector for that blockof data (i.e., index data portion), and (3) encrypt the data block(i.e., index data portion) with the new updated initialization vector.For the decryption step, in some instances, some of the data blocks mayhave already been decrypted from the index searching process. Tore-encrypt the data block, if nothing has changed (e.g., the content hasnot changed), the initialization vector can be changed so a differentencrypted data block with the same content is produced. An exampleprocess is to use an encryption process with a counter as aninitialization vector (e.g., increase the initialization vector everytime the data block is re-encrypted). In some instances, theinitialization vector can be combined with the underlying data such thatthe encrypted data includes both the underlying data and theinitialization vector. In other instances, the initialization vector canbe used to determine an encryption key used to encrypt the block ofdata. In such an instance, the initialization vectors for specific filescan be stored (e.g., at the security device 205) such that theappropriate initialization vector can be used to decrypt data whenretrieved. In either instance, the data uploaded to the backend storagecan appear different to the backend storage.

In some instances, to hide which logical memory blocks or whether anylogical memory blocks were used, the index file upload process cangather the data blocks that were downloaded and re-encrypted, and thenupload all of them (regardless of what data blocks have changed). Inother instances, only a portion of the data blocks downloaded areuploaded, such as those that have changed in value. In still otherinstances, a random subset of the data blocks that did not change isuploaded with the modified data blocks.

The index file can be updated periodically, automatically, or manuallyby commands (by a system user). For example, the steps of parsing 504,retrieving 505, and sending a signal to write 506 in the index fileupload process 500 can be performed periodically, repeatedly,automatically (e.g., every time the index file is updated, the indexfile can be uploaded according to the index file upload process), ormanually by commands (by a system user).

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Where methods and/or schematics described above indicatecertain events and/or flow patterns occurring in certain order, theordering of certain events and/or flow patterns may be modified. Whilethe embodiments have been particularly shown and described, it will beunderstood that various changes in form and details may be made.

For example, instead of using the pseudorandom number generator, in someinstances, a mapping table that maps the client document identifier tothe backend document identifier for the documents stored in thezero-trust storage can be used. Such a mapping table, however, does notscale as well as the system described above and the mapping table itselfstores more information regarding the file stored in the zero-truststorage than the system described above. Such a mapping table couldpotentially leak the actual document identifiers used to retrieve thedocuments stored in the zero-trust storage. The mapping table can alsobecome large both in memory and on disk. As such, the embodiment usingpseudorandom number generator is potentially more secure and more memoryefficient as there is no single catalog of the documents stored in thezero-trust storage.

While some steps in the processes and/or methods described with respectto the flow charts in FIGS. 3-5 refer are described to be performed by aspecific module (e.g., the randomizer module 213 of the security device205 or the zero-trust storage 101), in other embodiments, such steps canbe performed by any other suitable module(s), processor(s), device(s)and/or the like.

The above-disclosed systems and methods allow for fast and securesearching of encrypted documents. Specifically, by storing an index ofencrypted documents in a storage system as described herein (e.g., usingsecurity device 105 of FIG. 1), security around such an index can beincreased. For example, using a security module (e.g., security module206 of FIG. 2), such indexes can be randomly stored in a backendstorage. This also allows users to search encrypted documents usingsearch strings, rather than file identifiers increasing the security ofthe link between the data indexed in such files and the fileidentifiers. This can allow users to easily and securely searchencrypted documents stored with the index and/or in another storagelocation.

Although various embodiments have been described as having particularfeatures and/or combinations of components, other embodiments arepossible having a combination of any features and/or components from anyof embodiments as discussed above.

Some embodiments described herein relate to a computer storage productwith a non-transitory computer-readable medium (also can be referred toas a non-transitory processor-readable medium) having instructions orcomputer code thereon for performing various computer-implementedoperations. The computer-readable medium (or processor-readable medium)is non-transitory in the sense that it does not include transitorypropagating signals per se (e.g., a propagating electromagnetic wavecarrying information on a transmission medium such as space or a cable).The media and computer code (also can be referred to as code) may bethose designed and constructed for the specific purpose or purposes.Examples of non-transitory computer-readable media include, but are notlimited to, magnetic storage media such as hard disks, floppy disks, andmagnetic tape; optical storage media such as Compact Disc/Digital VideoDiscs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), andholographic devices; magneto-optical storage media such as opticaldisks; carrier wave signal processing modules; and hardware devices thatare specially configured to store and execute program code, such asApplication-Specific Integrated Circuits (ASICs), Programmable LogicDevices (PLDs), Read-Only Memory (ROM) and Random-Access Memory (RAM)devices. Other embodiments described herein relate to a computer programproduct, which can include, for example, the instructions and/orcomputer code discussed herein.

Some embodiments and/or methods described herein can be performed bysoftware (executed on hardware), hardware, or a combination thereof.Hardware modules may include, for example, a general-purpose processor,a field programmable gate array (FPGA), and/or an application specificintegrated circuit (ASIC). Software modules (executed on hardware) canbe expressed in a variety of software languages (e.g., computer code),including C, C++, Java™, Ruby, Visual Basic™, and/or otherobject-oriented, procedural, or other programming language anddevelopment tools. Examples of computer code include, but are notlimited to, micro-code or micro-instructions, machine instructions, suchas produced by a compiler, code used to produce a web service, and filescontaining higher-level instructions that are executed by a computerusing an interpreter. For example, embodiments may be implemented usingimperative programming languages (e.g., C, Fortran, etc.), functionalprogramming languages (Haskell, Erlang, etc.), logical programminglanguages (e.g., Prolog), object-oriented programming languages (e.g.,Java, C++, etc.) or other suitable programming languages and/ordevelopment tools. Additional examples of computer code include, but arenot limited to, control signals, encrypted code, and compressed code.

What is claimed is:
 1. A method for securely searching a database storedin a storage device, comprising: receiving, at a first compute deviceand from a second compute device, a search request including a searchstring associated with a characteristic in a set of electronic documentswithin the database; identifying, by the first compute device and basedon the search string, a set of pseudorandom logical block addressesassociated with a set of storage locations of the database by (1)identifying a seed based at least in part on the search string and (2)calculating the set of pseudorandom logical block addresses for thedatabase based on the seed; retrieving, by the first compute device,index data stored at each storage location from the set of storagelocations using the set of pseudorandom logical block addresses;compiling, by the first compute device, an electronic search result filebased on the index data stored at each storage location from the set ofstorage locations; identifying, by the first compute device and based onthe electronic search result file, a set of information associated withthe characteristic in the set of electronic documents and sending afirst signal that includes the set of information to the second computedevice; modifying, by the first compute device, an initialization vectorfor the index data stored at each storage location from the set ofstorage locations to define modified index data for each storagelocation from the set of storage locations; and sending, by the firstcompute device, a second signal to the database to write the modifiedindex data for each storage location from the set of storage locationsto that storage location from the set of storage locations within thedatabase.
 2. The method of claim 1, wherein the set of informationincludes at least one of a set of locations of the characteristic in theset of electronic documents or a set of statistical informationassociated with the characteristic in the set of electronic documents.3. The method of claim 1, wherein: the characteristic is a firstcharacteristic, and the set of information includes relationshipinformation associated with a relationship between the firstcharacteristic in the set of electronic documents and a secondcharacteristic in the set of electronic documents.
 4. The method ofclaim 1, further comprising: encrypting, by the first compute device,the modified index data for each storage location from the set ofstorage locations prior to sending the second signal to the database. 5.The method of claim 1, wherein the first signal includes a set ofdocument identifiers, each document identifier from the set of documentidentifiers associated with an electronic document from the set ofelectronic documents associated with the characteristic.
 6. The methodof claim 1, wherein the set of pseudorandom logical block addresses isassociated with the set of storage locations of a plurality of databasesincluding the database.
 7. The method of claim 1, wherein: the modifyingthe initialization vector includes generating a modified initializationvector for the index data stored at each storage location from the setof storage locations, the method further includes: determining, by thefirst compute device and based on the modified initialization vector, anencryption key for each storage location from the set of storagelocations, encrypting, by the first compute device, the modified indexdata for each storage location from the set of storage locations usingthe encryption key for that storage location from the set of storagelocations.
 8. The method of claim 1, wherein: the index data stored ateach storage location from the set of storage locations are encryptedindex data; the method further includes decrypting, by the first computedevice, the encrypted index data at each storage location from the setof storage locations prior to compiling the electronic search resultfile.
 9. The method of claim 1, wherein the set of pseudorandom logicalblock addresses is not stored in long-term memory.
 10. The method ofclaim 1, further comprising: determining, by the first compute deviceand based on the search string, a hash value using a hash function, theidentifying the seed includes generating the seed using the hash valueas an input to a seed mapping function.
 11. The method of claim 1,wherein the seed is a hash value, the hash value being determined basedon the search string using a hash function.
 12. A method for securelysearching a database stored in a storage device, comprising: receiving,at a first compute device and from a second compute device, a searchrequest including a search string associated with a characteristic in aset of electronic documents within the database; identifying, by thefirst compute device, a set of logical block addresses associated with aset of storage locations of the database using a cryptographicpseudorandom hash function with the search string as an input to thecryptographic pseudorandom hash function; retrieving, by the firstcompute device, index data stored at each storage location from the setof storage locations using the set of logical block addresses;compiling, by the first compute device, an electronic search result filebased on the index data stored at each storage location from the set ofstorage locations; identifying, by the first compute device and based onthe electronic search result file, a set of information associated withthe characteristic in the set of electronic documents and sending afirst signal that includes the set of information to the second computedevice; modifying, by the first compute device, an initialization vectorfor the index data stored at each storage location from the set ofstorage locations to define modified index data for each storagelocation from the set of storage locations; and sending, by the firstcompute device, a second signal to the database to write the modifiedindex data for each storage location from the set of storage locationsto that storage location from the set of storage locations within thedatabase.
 13. The method of claim 12, wherein the set of informationincludes at least one of a set of locations of the characteristic in theset of electronic documents or a set of statistical informationassociated with the characteristic in the set of electronic documents.14. The method of claim 12, wherein: the characteristic is a firstcharacteristic, and the set of information includes relationshipinformation associated with a relationship between the firstcharacteristic in the set of electronic documents and a secondcharacteristic in the set of electronic documents.
 15. A non-transitoryprocessor-readable medium storing code representing instructions to beexecuted by a processor for securely searching a database stored in astorage device, the code comprising code to cause the processor to:receive, from a compute device, a search request including a searchstring that corresponds to a characteristic in a set of electronicdocuments within the database; identify a seed based at least in part onthe search string; identify, based on the seed, a set of pseudorandomlogical block addresses associated with a set of storage locations ofthe database; retrieve index data stored at each storage location fromthe set of storage locations using the set of pseudorandom logical blockaddresses; compile an electronic search result file based on the indexdata stored at each storage location from the set of storage locations;send a first signal that includes the electronic search result file tothe compute device in response to the search request; define, inresponse to retrieving the index data, modified index data for eachstorage location from the set of storage locations based on the indexdata stored at each storage location from the set of storage locations;encrypt the modified index data for each storage location from the setof storage locations to define encrypted modified index data; and send asecond signal to the database to write the encrypted modified index datafor each storage location from the set of storage locations to thatstorage location from the set of storage locations within the database.16. The non-transitory processor-readable medium of claim 15, whereinthe set of pseudorandom logical block addresses is not stored inlong-term memory.
 17. The non-transitory processor-readable medium ofclaim 15, wherein the index data are encrypted index data.
 18. Thenon-transitory processor-readable medium of claim 15, further comprisingcode to cause the processor to: determine, based on the search string, ahash value using a hash function, the identifying the seed includesgenerating the seed using the hash value as an input to a seed mappingfunction.
 19. The non-transitory processor-readable medium of claim 15,wherein the first signal includes a set of document identifiers, eachdocument identifier from the set of document identifiers associated witha document from the set of electronic documents associated with thecharacteristic.
 20. The non-transitory processor-readable medium ofclaim 15, wherein the set of pseudorandom logical block addresses isassociated with the set of storage locations of a plurality of databasesincluding the database.